Disk Issues Lab

The lab in this module must be run on server1.example.com.

Goal
  • Recover corrupted LVM metadata

1. LVM Recovery

  1. Look at a list of current VGs, then create a backup of the LVM metadata for the extra volume.

    # vgs
    # vgcfgbackup extra
    # ls -al /etc/lvm/backup
  2. Create a VM snapshot if your hypervisor supports it.

  3. Simulate corruption of the LVM metadata on the extra disk, then reboot the system (never do this in practice!).

    # dd if=/dev/zero of=/dev/vdb1 count=10
  4. Check for the extra VG; it should no longer appear.

    # vgs
  5. Find the PV’s UUID and device name from /etc/lvm/backup/extra:

     physical_volumes {
                    pv0 {
                           id = "x8gEhX-zLGb-GUJg-MkqT-7wlr-FH3v-use23e"
                           device = "/dev/vdb1"    # Hint only
                           status = ["ALLOCATABLE"]
                           flags = []
                           dev_size = 39483392     # 18.8271 Gigabytes
                           pe_start = 2048
                           pe_count = 602  # 18.8125 Gigabytes
                    }
            }
  6. Before restoring the VG, you must reboot the system since the VG was removed uncleanly.

    # reboot
  7. Repair the PV’s metadata (ignore the error about not being able to find the device).

    # pvcreate --uuid <UUID FOR PV FOUND IN BACKUP> \
        --restorefile /etc/lvm/backup/extra /dev/<DEVICE FOUND IN BACKUP>
  8. Restore the VG, re-enable the VG, and then verify.

    # vgcfgrestore extra
    # vgchange -a y extra
    # vgs
    # mount /dev/extra/disk /mnt/temp
    # ls /mnt/temp
    # md5sum /etc/passwd /mnt/temp/passwd